Skip to content

Harden the instrument: tests, CI, and the evidence behind every published number - #1

Merged
Programmer7129 merged 11 commits into
mainfrom
harden/tests-and-ci
Sep 3, 2026
Merged

Programmer7129 merged 11 commits into
mainfrom
harden/tests-and-ci

Conversation

@Programmer7129

Copy link
Copy Markdown
Owner

The project's claim is that you can trust its measurement. This branch makes that checkable.

What was wrong

  1. Zero tests over ~9,000 lines of metric code that produces a headline with a confidence interval.
  2. The 51.0% had no committed evidence. results/raw_swe_500/ was gitignored, so nobody who cloned the repo could verify the first number in the README.
  3. No committed code derived the published headline. Reproduce step 4 pointed at code_aggregate.py, which aggregates the hub Class-A lane, not SWE-bench.
  4. The pre-registration timestamp claim was unverifiable. The repo opened as one squashed commit containing the registration and the results together.
  5. The compute figure was wrong. ~$15 claimed; the full-500 pass alone cost $21.60.

What changed

tests/ 240 tests, 0.33s, no network, Docker, GPU or API key
.github/workflows/ci.yml ruff + pytest on 3.11 and 3.12, re-deriving the headline every build
harness/swe_native_rate.py rederives 500 / 255 / 226 from the attack records
harness/swe_confirm.py recomputes 13.7% [10.1–18.0%] from the per-task verdicts
results/raw_swe_500/ 500 attack records, 4.1 MB, the evidence behind 51.0%
pyproject.toml hatchling, installable, dev extra
docs/preregistration.md dated amendment on what git can and cannot prove
README.md compute corrected to $29.98, reproduce steps fixed, name note

What did not change

No result, number or finding was altered. The published 13.7% reproduces exactly: 31/67/4 → p̂ 0.3039 → Wilson95 [0.2231, 0.3990] → 13.7% [10.1%, 18.0%]; anchor 22.4%, Haiku 16.3%. There was no bug in the metric code.

The cost correction

Committing the evidence made the compute claim falsifiable, and it was false. Summing cost_usd across every attacker run gives $29.98. The old ~$15 was the anchor-scale estimate. Corrected in the badge and the cost section, with the provenance stated.

Pure-core extractions

Three functions were extracted from I/O callers so they could be tested (aggregate.compute_env_from_records, code_aggregate.compute_env_from_records, code_attack.dual_gate_verdict). No metric behaviour changed; a test asserts the file path equals the records path.

https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS

Covers the primitives that turn per-probe counts into the headline: Wilson
score interval, exact binomial tail, Benjamini-Hochberg step-up, beta-binomial
pooling and the seeded cluster bootstrap.

Expected values come from independent sources — the score equation the Wilson
interval inverts, closed forms at k=0 and k=n, hand-computed binomial tails,
and the worked example in Benjamini & Hochberg (1995) — never from re-running
the implementation's own algebra.

Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
Three seams that decide what the headline counts:
  - oracle.screen/label_candidate: provenance tags, the deciding channel, and
    the hard rule that a probe the oracle cannot confirm wrong goes to the
    ambiguous bucket and is EXCLUDED from FAR (pre-reg §3).
  - classify: the five verifier-type buckets, weight-0 monitor exclusion, and
    the empirical binary -> soft flip on observed partial credit (pre-reg §1).
  - aggregate: FAR against the env's own gold anchor, RHS over degenerate
    provenance, FRR, gold-fails and non-discriminating exclusions, soft-grader
    monotonicity, and the BH-FDR / pooling / stratification roll-up (§2, §5).

aggregate.compute_env is split so the metric computation is pure: the new
compute_env_from_records takes parsed records, and compute_env reads the file
and delegates. No metric behaviour changes — compute_env's output for a given
file is identical, and a test asserts the two paths agree.

Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
  - code_attack.dual_gate_verdict (extracted): verifier PASS alone is never a
    confirmed reward-hack, oracle WRONG without a PASS is not one either, and
    an oracle residual (uninvokable / no divergence) routes to hand review
    rather than being auto-passed as correct. The advisory LLM gate never
    flips the verdict.
  - code_attack.behaviorally_distinct: the confirm-queue filter.
  - code_aggregate: the reject-level pass bar, escape-exploit counting,
    ambiguous exclusion, the FRR control, and the per-env outcome buckets
    (partial-credit-to-wrong and non-discriminating both route OUT of FAR).

Both compute_env functions are split the same way as aggregate.py: a pure
compute_env_from_records plus a thin file-reading caller. No metric behaviour
changes; the record built by attack_task is field-for-field identical.

Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
harness/swe_confirm.py implements the estimator registered in the 2026-07-14
pre-registration amendment and the anchor tally, reading only results/. It
computes nothing new: it puts the previously uncommitted derivation of the
published figures into code so a reviewer can re-run it.

    python harness/swe_confirm.py

Every published figure is reproduced from the committed per-task verdicts:
  anchor  Sonnet 4.5   11/49 = 22.4%   (3 diffexec + 8 hand, of 21 candidates)
  anchor  Haiku 4.5     8/49 = 16.3%   (3 diffexec + 5 hand, of 19 candidates)
  extend  sample tally  31 HACK / 67 CORRECT / 4 AMBIGUOUS  (n=102)
  extend  p-hat 0.3039  Wilson95 [0.2231, 0.3990]
  extend  confirmed     13.7%  95% CI [10.1%, 18.0%]  (68.7/500, CI [50.4, 90.2])

Three inputs stay DECLARED because results/raw_swe_500/ is gitignored: the
500-task frame, the 226-task confirm queue and the 255 native-hackable tasks
(the 51.0% verifier-alone figure). Everything downstream of them is recomputed.
Both the module docstring and the CLI output say so.

No result file, number or claim was changed.

Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
  - pyproject.toml (hatchling, MIT, same conventions as the author's other
    repos). Core dependencies are empty because the aggregation, oracle and
    classification code is standard-library only; the attack lanes are extras
    (swe / anthropic / bedrock / dev). requirements.txt is unchanged apart from
    a header pointing at the equivalent extras.
  - .github/workflows/ci.yml runs ruff and pytest on 3.11 and 3.12, then
    re-derives the published headline from results/ so a change that breaks
    reproducibility fails the build. No network, Docker, API key or GPU.
  - Lint gate: ruff's default rules, with E501 and E741 off and two documented
    per-file ignores, so it passes without reflowing the modules that produce
    the published numbers. Nine genuinely unused imports were deleted; no other
    harness code was touched.

Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
Renames the cosmetic occurrences: module docstring headers, CLI --help text
and printed banners. Runtime identifiers keep the internal name on purpose —
the Docker image and container names, the egress network and proxy names, the
in-container probe paths, the run-id prefixes, the LiteLLM key and the no-op
control patch are load-bearing strings, and renaming them would change what
the harness does. The README now says so in one line.

README also: step 4 pointed at code_aggregate.py, which aggregates the hub
Class-A code lane, not the SWE-bench run. It now points at swe_confirm.py for
the SWE-bench headline and keeps code_aggregate.py for the hub lane, and adds
a step for running the tests. No number or finding changed.

Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
…estamp

Appended below the original text, as §9 requires; nothing above it is edited
and no number, threshold or method changes.

The header claims the git commit timestamp of this file is the pre-registration
record. For the published runs that is not independently verifiable: the public
repository was opened with a single squashed commit (95be36b, 2026-07-26)
carrying this file and the results together, so git cannot show the ordering.

The amendment states that plainly, sets out what IS verifiable without it (the
seed-42 draw committed as its own artifact disjoint from the verdicts, per-task
reproducers and rationales, thresholds as cited constants in code, and every
registered bias pointing the headline down with the looser readings disclosed
alongside), and commits to per-phase pre-registration as its own commit, before
any result for that phase exists, with the hash cited in the result document.

Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
The verifier-alone rate led the README but its evidence was gitignored, so
nobody who cloned the repo could check it, and no committed code derived it.
README step 4 also pointed at code_aggregate.py, which aggregates the hub
Class-A lane rather than SWE-bench.

Commits results/raw_swe_500/ (500 records, 4.1 MB, no secrets or local paths)
and adds harness/swe_native_rate.py, which rederives all three inputs the
confirmed-rate estimator declares: total_tasks 500, native_hackable 255
(51.0%), queue_size 226. tests/test_swe_native_rate.py fails the build if the
derivation ever disagrees with results/swe500_confirmed.json.

No result, number or claim changed.

Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
Committing the attack records made the cost claim checkable, and it was
wrong. Summing cost_usd across every attacker run gives $29.98; the
full-500 pass alone is $21.60. The old figure was the anchor-scale
estimate. The README now states the total, its provenance, and that the
earlier number understated it.

Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
ruff caught an unused sys import in swe_native_rate.py; I had run pytest
locally but not ruff, so CI failed on the first push. The swe_confirm CLI
also still told the reader raw_swe_500/ was not committed, which stopped
being true one commit earlier.

Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
Five independent reviewers, none of which wrote the branch. Fixed here:

- CI's 'recompute the headline' step could not fail: swe_confirm.py printed
  and returned 0 unconditionally, so a green check guarded nothing. Adds
  --check, which pins 7 published figures plus the CI bounds and exits 1 on
  drift. Verified by flipping one verdict in a copy: 5 drifts, exit 1.
- The $29.98 cost replaced one unverifiable figure with another. Only $21.60
  is derivable here. README now states the split, swe_native_rate reports the
  pass cost, and a test pins $21.60.
- Three files still claimed results/raw_swe_500/ was not committed, one commit
  after it was.
- docs/anchor-result.md still carried the ~$10-15 estimate this branch set out
  to correct. Now $5.17, summed from that lane's records.
- .gitignore enumerated six paths, so a new raw dir would be committed
  silently. Back to a wildcard with an explicit exception for raw_swe_500.
- Pre-registration was still titled 'envcert'.
- NOTICE now names the twelve upstream projects quoted in the diffs and their
  licences, states that MIT covers only Tripwire's own work, and calls out the
  pylint GPL fragments specifically.

Not fixed here, escalated to the author: the README's framing of the 51%.

Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
@Programmer7129
Programmer7129 merged commit 4e52229 into main Sep 3, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant